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Abstract 

Methodological reviews, reviews that concentrate on research methods rather than research 
outcomes, have been used in a variety of fields to improve research practice, inform debate, and 
identify islands of practice. In this article, we report on the results of a methodological review of 
all of the articles published in Georgia Educational Researcher from 2003-2010. We examined 
the methodological characteristics, authorial characteristics, and methodological quality of those 
articles using quantitative content analysis. The major findings were that the (a) proportions of 
the type and traditions of articles published in Georgia Educational Researcher were similar to 
the proportions in education research articles in general, (b) case study research and correlational 
research were most prominent, (c) a few universities accounted for most of the articles published, 
(d) male and females were published in equitable proportions, and (e) there were no statistically 
significant differences in methodological quality between first author’s gender and among 
affiliations, type of research, or year of publication. We interpret these last two findings to be 
evidence of a lack of editorial bias. A few minor suggestions for improving the quality of 
qualitative research articles are given. 

Keywords: Methodological review, methodological quality, systematic review 
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A Methodological Review of the Articles Published in 
Georgia Educational Researcher from 2003-2010 

Methodological reviews, reviews that focus on research methods rather than research 
outcomes, have been used in many fields to improve research practice, inform debate, and 
identify islands of practice. For example, Keselman et al. (1988) conducted a methodological 
review of education researchers' statistical practices. Subsequently, that work helped inform the 
guidelines of the APA Task Force on Statistical Inference's influential report Statistical Methods 
in Psychology Journals: Guidelines and Explanations (Wilkinson & APA Task Force on 
Statistical Inference, 1999), which went on to inform numerous authors, editors, and reviewers in 
best statistical reporting practice. In addition, the Social Science Research Council and the 
National Academy of Education’s Joint Committee on Educational Research documented a need 
for “. . . data and analysis of the research enterprise, .... determination of where education 
research is conducted and by whom, [and] .... identification of the range of problems addressed 
and the methods used to address them” (Ranis & Walters, 2004, pp. 798-799). 

The Georgia Educational Researcher (GER) is now in its eighth year of publication and 
has not had a methodological review published about it thus far. Therefore, the purpose of this 
research was to identify and quantify the types of articles published in GER and to review the 
methods used in the qualitative, quantitative, and mixed- methods studies. The expected benefits 
of this study include improved research practice through the identification of methodological 
strengths and weakness and improved understanding of the “trends, tribes, and territories” of 
Georgia educational researchers. The target audiences for this article are the readers of GER, 
GER editors and reviewers, and authors planning on submitting manuscripts to GER. 

The research questions that we strove to answer are listed below: 
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1 . What are the overall methodological characteristics of the articles published in GER? 

2. What are the overall authorial characteristics of the articles published in GER? 

3. What are the characteristics of the methodological quality of the articles published in 
GER? 

4. What are the predictors, if any, of the methodological quality of the articles published in 
GER? 

In the following sections of this manuscript, we discuss related methodological reviews, the 
methods we used to carry out this investigation, the results, and a discussion of our findings. 

Related Research 

While there have been many methodological reviews of the educational research 
literature over the years (e.g., Keselman et ah, 1999; Randolph, 2008), we have chosen to 
concentrate on what we consider to be the most comprehensive review to date — Gorard and 
Taylor (2004). In that review, Gorard and Taylor reviewed a representative sample of 94 articles 
from leading education research journals. They further validated their findings with 

• interviews with key stakeholders from across the education field;including researchers, 
practitioner representatives, policy makers and policy implementers; 

• a large-scale survey of the current methodological expertise and future training needs of 
UK education researchers; [and a] 

• detailed analysis and breakdown of 2001 RAE [Research Assessment Exercise, 2001]. (p. 
114) 

The findings from the Gorard and Taylor (2004) study that are most relevant to the 
current study relate to the proportions of articles that were classified as (a) empirical research 
with human participants and (b) nonempirical research (such as theoretical articles) or secondary 
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research (such as literature reviews). Also of relevance are the proportions of articles that were 
classified as quantitative, qualitative, or mixed methods. We will use those proportions as a point 
of comparison for the proportions we found in GER articles. Gorard and Taylor’s proportions 
are presented in Table 1. In essence, in education research articles in general over 80% of 
published articles reported on empirical research with human participants. Of those articles, there 
were equal amounts of quantitative and qualitative articles and there were very few mixed- 
methods articles. 

Table 1 

Characteristics of Education Research Articles from the Gorard and Taylor Review 



Characteristic 


Count 


Percent 


Type of article 


Empirical research with human participants 


79 


84 


Nonempirical research or secondary research 


15 


16 


Tradition of article, if empirical 


Quantitative 


43 


54 


Qualitative 


32 


41 


Mixed 


4 


5 



While it has been shown that these proportions can vary over subdisciplines of education 
and even over region of the first author's affiliation (see Randolph, 2008), we assume that these 
proportions are representative of the education research literature in the mid 2000s and across 
geographic areas. Therefore, we will use these proportions as a point of reference to compare to 
our own. 

Methods 

Neuendorfs (2002) method of quantitative content analysis was used in this review. That 
method consists of the following steps: 

1 . Developing a theory and rationale. 
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2. Conceptualizing variables. 

3. Operationalizing measures. 

4. Developing a coding form and coding book. 

5. Sampling. 

6. Training and determining pilot reliabilities. 

7. Coding. 

8. Calculating final reliabilities. 

9. Analyzing and reporting data. 

In the remainder of this section, we will provide information on sampling, training and 
determining pilot reliabilities, coding, calculating final reliabilities, and analyzing and reporting 
data. Because a pre-existing coding form was used, we do not report on the first four steps of 
Neuendorf’s method; however, we do provide information on the coding forms used. 

Coding forms 

Two coding forms for reviewing the methodological qualities of quantitative and 
qualitative articles were used; they are presented in Appendices A and B. They were adapted 
from Creswell (2012, p. 291 & p. 292, respectively). The quantitative coding form originally 
contained 44 questions in areas related to (a) the title of the study; (b) the problem statement; (c) 
review of the literature; (d) purpose, hypotheses, and research questions; (e) data collection; (f) 
data analyses and results; (g) writing; (h) internal validity; and (i) external validity. However, 
because of poor reliabilities, the internal validity items were not analyzed in this manuscript. See 
the section on interrater reliability for more information. In all of the items except for the items 
dealing with external validity, the raters were asked to indicate whether they agreed or disagreed 
with the item. In the external validity section, raters were asked to write in short answers. The 
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qualitative coding form had 32 items in the areas of (a) title for the study, (b) problem statement, 
(c) review of the literature, (d) purpose and research questions, (e) data collection, (f) data 
analysis and findings, and (g) writing. 

Sampling 

The sample for this study included all of the GER articles published from Volume 1, 

Issue 1 in the fall of 2003 to Volume 8, Issue 1 in the spring of 2010. (Technically, it was a 
census and not a sample because the universe of articles was reviewed.) In total, 42 articles were 
included in this review. Transcriptions of key note addresses were not included in this review. 
Training Raters and Determining Pilot Reliabilities 

The raters for this review were 15 students in an introductory educational research course 
in a doctoral program in curriculum and instruction at Mercer University. During the course, the 
raters were given approximately 45 hours of instruction over an eight- week period in content 
matter related to the items on quantitative and qualitative coding forms. Explicit instruction was 
also given on how to use the coding forms to code the articles. To determine pilot reliabilities, 
each rater was assigned one or more quantitative or qualitative articles. A mixed-methods article 
was counted as both a quantitative and qualitative article. After initially reviewing the articles, 
the raters and instructor came together to discuss and clarify the coding form items on which 
there was confusion. 

Calculating Final Reliabilities 

Raters were randomly assigned a set of quantitative and qualitative articles. In addition 
to the articles that the raters individually rated, there was one common article (either quantitative 
or qualitative) that multiple raters coded to assess the interrater reliability of their ratings. The 




METHODOLOGICAL REVIEW OE GER ARTTICLES 



measure of interrater reliability used in this article was a multirater variation of Bennet et al.’s 
free-marginal kappa statistic (See Randolph, 2005, and Warrens, 2010). 

Data Collection and Analyses 

Each rater was randomly assigned a set of one or more articles to rate and used the 
quantitative and/or qualitative coding forms to code the data. Mixed-methods articles were coded 
on both their quantitative and qualitative characteristics. 

Descriptive statistics were calculated for the articles’ authorial and methodological 
characteristics. To compare the proportions found in this article with the proportions found in the 
Gorard and Taylor (2004) review, x2 analyses were used. 

Factorial ANCOVA, using the technique described in Field (2009), was used to 
determine the predictors of methodological quality for qualitative articles. All of the assumptions 
for factorial ANCOVA had been met; namely, a visual analysis of a residual plot showed 
heteroscedasticity and had no influential data points. Also, Fevene’s test indicated homogeneity 
of variances, F(8, 2) = 0.55, p = .778. Qualitative quality was used as the outcome variable. 
Affiliation, qualitative tradition, and gender were used as fixed factors and year of publication 
was used a covariate. 

The data for quantitative quality did not meet the assumptions for factorial ANCOVA; 
therefore, nonparametric statistics were used instead. The Mann-Whitney test was used to 
compare quality between genders of first authors. The Kruskal- Wallis test was used to examine 
differences among groups on the affiliation, type of article, and year of publication variables. 
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Results 

Interrater Reliabilities 

Nine raters were randomly assigned to rate the same qualitative article using the 
qualitative coding form, which had 30 items, each of which had two categories. The nine raters’ 
percent of overall agreement on those 30 items was 83% and the kappa value was .63, indicating 
fair agreement above chance. A rule of thumb is that values of kappa above .70 indicate good 
agreement (Neuendorf, 2002). 

Nine raters were randomly assigned to read the same quantitative article, using the 
quantitative coding form, which had 36 items, each of which had two categories. Of those nine 
raters, only four completed their ratings without missing data. Since kappa cannot be calculated 
with missing data, those raters’ ratings were not included. Of those four raters, there was 100% 
agreement on those four items, which equates to a kappa value of 1.00. When including two 
raters who were only missing one or two items and then calculating kappa and percent of overall 
agreement per item then averaging the values, the value of kappa was .90 and the percent of 
overall agreement was 95%. 

One complication was that there were eight items on the quantitative coding form that 
dealt with internal validity and were applicable to experimental articles only; however, raters 
sometimes completed ratings of internal validity for correlational and descriptive articles as well. 
This demonstrates a serious lack of reliability on those variables, so an analysis of the internal 
validity of the quantitative articles will be excluded from this manuscript. 

Methodological Characteristics 

Table 2 below presents the methodological characteristics of the 34 empirical GER 
articles included in this review. Note that the six mixed-methods articles were coded using both 
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the quantitative and qualitative coding forms, so quantitative coding was done on 23 articles and 
qualitative coding was done on 17 articles. 

Table 2 

Methodological Characteristics of GER articles 



Methodological Characteristic 


Count 


% 


Type 


Empirical 


34 


81 


Eiterature Review 


2 


5 


Other 


6 


14 


If empirical, what tradition? 


Quantitative 


17 


50 


Qualitative 


11 


32 


Mixed 


6 


18 


If qualitative, what tradition? 


Case Study 


5 


39 


Ethnography 


1 


7 


Phenomenology 


3 


23 


Grounded Theory 


2 


15 


Narrative/ Biography 


2 


15 


(Could not be determined) 


(4) 


(N/A) 


If quantitative, what type? 


Experimental 


7 


30 


Correlational 


11 


48 


Descriptive 


5 


22 



In terms of the overall type of article, the overwhelming majority of articles were 
empirical (81%). There was not a statistically difference in the proportion of empirical GER 
articles and the proportion of empirical education research articles reviewed by Gorard and 
Taylor (2004), y2{2) = 0.20, p = .657. Of the empirical articles, quantitative articles were most 
prominent, followed by qualitative articles, then mixed-methods articles. 

In terms of the tradition used, the proportion of quantitative, qualitative, and mixed- 
methods GER articles was not statistically different from the proportions in the Gorard and 
Taylor review, x^(2) = 4.76, p = .093. Although there was not a statistically significant 
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difference at the .05 a level, mixed-methods articles composed 18% of GER articles, while 
mixed- methods articles only composed 5% of the articles in the Gorard and Taylor review. 

In terms of the qualitative tradition used, case study research was the most commonly 
used qualitative tradition in GER articles. In three of the qualitative articles, we were not able to 
determine what qualitative tradition was used. We consider this to be an indicator of poor 
methodological quality in these articles. 

Einally, in terms of the type of quantitative method used, articles that used a correlational 
method were most often published, followed closely by experimental articles. Purely descriptive 
quantitative articles, such as reports of survey findings, were used less frequently. 

Types of participants examined. Elementary students were the types of participants 
most frequently examined. Those were followed by high school students and school 
administrators. After school administrators, school staff and preservice teachers were most 
frequently examined. Parents, college professors who taught online classes, and student support 
personnel each had one article in which they were the primary participants. One article had 
mixed participants — administrators and support staff. 

Types of interventions examined. Interventions that dealt with reading literacy were 
examined most frequently. After literacy, interventions dealing with teacher certification and 
leadership were examined most frequently. 

Types of outcomes examined. Since reading interventions were the most frequently used 
types of intervention in GER articles, it is no surprise that the most frequently used outcomes 
were measures of reading fluency. The other outcomes in order of frequency were academic 
achievement, faculty perceptions, and attendance. The outcomes that were only used in one 
article are not listed here. The outcome could not be determined in two articles. 
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Types of settings examined. The research done in the GER articles occurred in a variety 
of settings and grade levels. Of the articles in which the authors mentioned their setting, five 
were set in urban schools, two in rural schools, and two in suburban schools. 

Authorial Characteristics 

Table 3 presents information on the gender of the first author, the first author’s affiliation, 
and the number of GER articles published by year. Note that these data only represent the 34 
empirical articles. The other category in the affiliation section included all of the universities that 
only contributed one article. In summary, there was about an equal percentage of female and 
male authors; the first author’s first affiliations were most often Valdosta State University, 
Georgia Southern University, and Georgia State University; and the number of articles published 
per year was more or less constant with an increase in 2010. 

Table 3 

Authorial Characteristics of GER Articles 



Characteristic 


Count 


% 


Gender 


Female 


15 


48 


Male 


16 


52 


(Indeterminate) 


(4) 




Affiliation 


Valdosta State University 


8 


24 


Georgia State University 


7 


21 


Georgia Southern University 


5 


15 


Other 


14 


41 


Year 


2003 


5 


15 


2004 


3 


9 


2005 


2 


6 


2006 


5 


15 


2007 


4 


12 


2008 


3 


9 


2009 


5 


15 


2010 


7 


21 
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Methodological Quality of Quantitative Articles 

Table 4 presents the results of each item on the quantitative coding form, which is 
included in Appendix A. The average of all of the items across the 17 quantitative and six mixed 
methods articles was 82.14% with 95% CIs of was 74.78 and 89.50. 

Table 4 



Ratings for Each Item on the Quantitative Coding Form 



Item 


% Yes (Count) 


95% CIs 


1 


81 (17) 


[63, 99] 


2 


82(17) 


[64, 99] 


3 


59(13) 


[37, 81] 


4 


95 (21) 


[86, 100] 


5 


91 (20) 


[78, 100] 


6 


95 (21) 


[86, 100] 


7 


100 (21) 


[86, 100] 


8 


88(15) 


[69, 100] 


9 


75 (14) 


[51, 99] 


10 


88(18) 


[69, 100] 


11 


71(16) 


[50, 92] 


12 


76(17) 


[56, 96] 


13 


62(13) 


[39, 85] 


14 


81 (18) 


[63, 99] 


15 


76(16) 


[56, 96] 


16 


85(17) 


[68, 100] 


17 


75(17) 


[54, 96] 


18 


70(15) 


[48, 92] 


19 


80(18) 


[61, 99] 


20 


86(18) 


[69, 100] 


21 


91 (10) 


[71, 100] 


22 


92 (21) 


[73, 100] 


23 


67(16) 


[35, 98] 


24 


75(17) 


[46, 100] 


25 


92 (21) 


[73, 100] 


26 


67(15) 


[45, 89] 


27 


90 (21) 


[77, 100] 


28 


67(16) 


[45, 89] 


29 


90 (20) 


[77, 100] 


30 


86(19) 


[69, 100] 


31 


95 (21) 


[86, 100] 


32 


100 (21) 


- 



Note. We used 100 as the upper bound of the confidence interval. 
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Figure 1 shows the means and 95% confidence intervals when the items were averaged 
within sections. All sections except for one seemed to have more or less the same proportion of 
quantitative quality scores; the quality of problem statements in quantitative and mixed-methods 
GER articles stood out above the other sections. 




Figure Average methodological quality of quantitative articles by section. 

Methodological Quality of Qualitative Articles 

Table 5 presents the ratings for each item on the qualitative coding form. Averaging all of 
the items together yielded a mean qualitative quality score of 77.82 with 95% confidence 



intervals of 69.67 and 85.97. 
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Table 5 



Ratings for Each Item on the Qualitative Coding Form 



Item 


% Yes (Count) 


95% CIs 


1 


94 


(16) 


[82, 


100] 


2 


35 


(6) 


[10, 


61] 


3 


94 


(17) 


[83, 


100] 


4 


100 


(17) 


- 


- 


5 


89 


(16) 


[73, 


100] 


6 


94 


(17) 


[83, 


100] 


7 


88 


(14) 


[69, 


100] 


8 


89 


(16) 


[73, 


100] 


9 


39 


(7) 


[14, 


64] 


10 


83 


(15) 


[64, 


100] 


11 


72 


(13) 


[49, 


95] 


12 


61 


(11) 


[36, 


86] 


13 


35 


(6) 


[10, 


61] 


14 


76 


(13) 


[54, 


99] 


15 


75 


(12) 


[51, 


99] 


16 


78 


(14) 


[57, 


99] 


17 


67 


(12) 


[43, 


91] 


18 


89 


(16) 


[73, 


100] 


19 


83 


(15) 


[64, 


100] 


20 


94 


(17) 


[83, 


100] 


21 


83 


(15) 


[64, 


100] 


22 


83 


(15) 


[64, 


100] 


23 


67 


(12) 


[43, 


91] 


24 


82 


(14) 


[62, 


100] 


25 


100 


(17) 


- 


- 


26 


88 


(15) 


[71, 


100] 


27 


88 


(15) 


[71, 


100] 


28 


47 


(8) 


[21, 


74] 


29 


47 


(8) 


[21, 


74] 


30 


100 


(16) 


- 


- 


Note. We used 100 as the upper bound of the confidence interval. 

Figure 2 shows the average qualitative methodology quality scores by each section on the 
qualitative coding form. Similar to the results show in Figure 1, all of the sections have more or 
less the same quality scores except for the problem statement, which stands out above the other 



sections. 
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Figure 2. Average methodological quality of qualitative articles by section. 

Predictors of Methodological Quality 

In terms of quantitative methodological quality, the results indicated that neither gender, 
U (21) = 53.50, z = -.04,/) = .972; nor affiliation, 7/(3) = 3.91,/) = 2.71; nor type of article (i.e., 
experimental, correlational, or descriptive), 7/(2) = 2.15, p = 2.51; nor year of publication, 77(7) 
= 7.42, p = .386, were statistically significant predictors of methodological quality. Similarly, 
there were no statistically significant predictors of the methodological quality of the qualitative 
articles either; gender, 7^(1, 2) = 0.12, p = .486, r\ = .26; affiliation, F{2, 2) = 0.12, p = .891, = 

.11; qualitative tradition, F(3, 2) = 0.49, p = .124, rf = .42; year of publication, 7^(1, 2) = 0.12, p 
= .760, = .06. 
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Discussion 

Summary of Results 

Methodological Characteristics. In terms of methodological characteristics, the 
proportion of types and traditions of articles in GER was very similar to the proportion of types 
and traditions of articles in education research articles in general, if we assume that the Gorard 
and Taylor (2004) review is representative of education research articles in general. The majority 
of articles reported on empirical research with human participants. Of those articles, quantitative 
articles made up the majority, followed by quantitative and mixed-methods articles, in that order. 
Although the difference was not statistically significant, there was a 13% difference in the 
percentage of mixed-methods articles between GER articles and education research articles in 
general. It is difficult to determine if this is because the Gorard and Taylor review was conducted 
in 2004 and mixed-methods articles have gained popularity since then or if the difference is a 
characteristic of GER. Of the quantitative articles, correlational research was most prevalent, 
followed by experimental and descriptive research. Of the qualitative articles, case study 
research was the most prevalent qualitative tradition used. In many of the qualitative articles, the 
qualitative tradition could not be determined. 

Authorial characteristics. In terms of the affiliations of first authors, Valdosta State 
University is the current host institution of GER, so it is no surprise to use that it is also the 
institution with the most GER articles. The institution who is represented in GER second most 
frequently is Georgia Southern University; we hypothesize that this is the case because Georgia 
Southern University hosts the GERA conference, which is often a source of GER articles. 

In terms of gender, there was about an equal number of male and female first authors in 
GER articles. Although we could not find a reliable statistic indicating the percentage of female 
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education professors in Georgia, we were able to find a reliable statistic indicating that, in 2009, 
46% of professors in the U.S. were female (United Nations Statistics Division, 2011). A 
binomial test indicated that there was not a statistically significant difference in the proportion of 
female first authors of GER articles (48.39%) and the proportion of female professors in the 
United States (46.00%), p = .464. Therefore, we take this to be evidence of a lack of editorial 
gender bias. The number of articles published each year has stayed steady from 2003 to 2010, 
with the exception of a small spike in 2010. 

Methodological quality. Using the adaptations of Creswell’s (2012) checklists for 
evaluating quantitative and qualitative research revealed some strengths and weakness in the 
GER articles. Eor both quantitative and qualitative articles, problem statements were the sections 
that had the highest ratings of methodological quality. While there were no areas that particularly 
stood out as weaknesses in the quantitative articles, the purpose statements and research 
questions of qualitative articles stood out as an area that received low ratings. Another area of 
weakness in the qualitative articles was that the authors did not make explicit which qualitative 
tradition they are adhering to. 

Methodological quality was about the same between genders, affiliation, year of 
publication, and type of article. We take this to be an indicator that the editors and reviewers of 
GER apply their editorial standards equally and fairly. 

Recommendations 

Methodological reviews can be especially impactful when it is found that research 
practice deviates sharply from best research practice. This was not the case here. Therefore, the 
only recommendations we have to improve practice are few and minor. Eirst, we believe that 
authors of qualitative research should explicitly state what qualitative tradition they are using. 
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Second, Creswell (2012) has scripts for writing high-quality qualitative research questions and 
purpose statements and we believe that it would be beneficial for authors to use those scripts to 
clarify their questions. Of course, we acknowledge that Creswell’s criteria for evaluating 
qualitative and quantitative are just one of many sets of valid criteria for the evaluation of 
methodological quality. 
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Appendix A: Coding Form for Quantitative Studies (Adapted from Creswell, 2012) 



Check Checklist for Evaluating the Process of a Quantitative Study 

If yes 


Title for the Study 


1 Does it reflect the major independent and dependent variables? 


2 Does it express either a comparison among groups or a relationship among variables? 


3 Does it convey the participants and site for the study? 


Problem Statement 


4 Does it indicate an educational issue to study? 


5 Has the author provided evidence that this issue is important? 


6 Is there some indication that the author located this issue through a search of past literature or from personal experiences? 


7 Does the research problem fit a quantitative approach? 


8 Are the assumptions of the study consistent with an approach? 


Review of the Literature 


9 Are the studies about the independent and dependent variables clearly reviewed? 


10 Does the review end with how the author will extend or expand the current body of literature? 


1 1 Does the study follow APA style? 


Purpose, Hypotheses, and Research Questions 


12 Does the author specify a purpose statement? 


13 Is the purpose statement clear, and does it indicate the variables, their relationship, and the people and site to be studied? 


14 Are either hypotheses or research questions written? 


15 Do these hypotheses or questions indicate the major variables and the participants in the study? 


16 Do the purpose statement and hypotheses or research questions contain the major components that will help a reader 
understand the study? 


17 Has the author identified a theory or explanation for the hypotheses or questions? 


Data Collection 


18 Does the author mention the steps taken to obtain access to people and sites? 


19 Has the author identified good, valid, and reliable instruments to use to measure the variables? 


20 Are the instmments administered so that bias and error are not introduced into the study? 


Data Analysis and Results 


21 Are the statistics chosen for analysis consistent with the research questions hypotheses, variables, and scales of measurement? 


22 Is the unit of analysis appropriate to address the research problem? 


23 Are the data adequately represented in tables and figures? 


24 Do the results answer the research questions and address the research problem? 


25 Are the results substantiated by evidence? 


26 Are generalizations from the results limited to the population of participants in the study? 


Writing 


27 Is the structure of the overall study consistent with the topics addressed in a quantitative study? 


28 Are educational and social science terms carefully defined? 


29 Are variables labeled in a consistent way throughout the study? 


30 Is the study written using extensive references? 


3 1 Is the study written using an impersonal point of view? 


32 Is the study written appropriately for intended audiences? 


External Validity 


33 What units does this study generalize to? 


34 What treatments does this study generalize to? 


35 What outcomes does this study generalize to? 


36 What settings does this study generalize to? 
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Appendix B: Coding Form for Qualitative Studies (adapted from Creswell, 2012) 



Checklist for Evaluating the Process of a Qualitative Study (Check if yes) 


Title for the Study 


1 Does it reflect the central phenomenon being studied? 


2 Does it reflect the people and site being studied? 


Problem Statement 


3 Does it indicate an education issue to study? 


4 Has the author provided evidence that the issue is important? 


5 Is there some indication that the author located this issue through a search of past literature or from 
personal experience? 


6 Does the research problem fit a qualitative approach? 


7 Are the assumptions of the study consistent with a qualitative approach? 


Review of the literature 


8 Has the author provided a literature review of the research problem under study? 


9 Has the author signaled that the literature review is preliminary of tentatively based on the findings in 
the study? 


10 Does the study follow APA style? 


Purpose and Research Questions 


1 1 Does the author specify both a purpose statement and a central research question? 


12 Do the purpose statement and central question indicate the central phenomenon of the study and the 
people and place where the study will occur? 


13 Are subquestions written to narrow the central question to topic area or foreshadow the steps in data 
analysis? 


Data collection 


14 Has the author taken steps to obtain access to people and sites? 


15 Has the author chosen a specific purposeful sampling strategy for individuals or sites? 


16 Is the data collection clearly specified and is it extensive? 


17 Is there evidence that the author has used a protocol for recording data? 


Data Analysis and Findings 


18 Were appropriate steps taken to analyze the text or visual data into themes, perspectives, or categories? 


19 Was sufficient evidence obtained (including quotes) to support each theme or category? 


20 Were multiple -layer themes or categories derived? 


21 Did the findings answer the research question? 


22 Were the findings realistic and accurate? 


23 Were steps taken to support this conclusion through verification? 


24 Were the findings represented in themes or categories so that multiple perspectives can be easily seen? 


25 Were the findings represented in narrative discussions or in visuals? 


Writing 


26 Was the account written persuasively and convincingly? 


27 Was the overall account consistent with one of the many forms for presenting qualitative research? 


28 Was the account written to include literary approaches, such as the use of metaphor, surprises, detail, 
dialogue, and complexity? 


29 Was it written using a personal point of view? 


30 Is the study written appropriately for the intended audiences? 





